Skip to content

feat(orchestrator): route per-queue storage through host profiles and fix the buildsignal republish dedup - #538

Merged
behinddwalls merged 1 commit into
mainfrom
preetam/storage-profile-routing
Aug 6, 2026
Merged

feat(orchestrator): route per-queue storage through host profiles and fix the buildsignal republish dedup#538
behinddwalls merged 1 commit into
mainfrom
preetam/storage-profile-routing

Conversation

@behinddwalls

@behinddwalls behinddwalls commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

Why?

Per-queue backend routing is host policy: profiles.go already routes every other seam (build runner, change provider, analyzer) per queue, so storage joins the same registry now that its contract is factory-resolved. Separately, end-to-end validation caught a regression from the payload change: repointing buildsignal's speculate publish from the batch-ID partition to the queue partition made its message share the (topic, partition, message-id) triple with the batch controller's original speculate publish, and the MySQL queue backend dedupes publishes on that triple against rows not yet garbage-collected — so the "build finished, re-evaluate" nudge was silently dropped and the batch stalled in Speculating.

What?

Profile gains a per-queue storage factory entry; every profile points at the shared MySQL backend by default and Profiles.StorageFactory() routes each resolution through the queue's profile, so a deployment that splits queues across backends overrides one field. The orchestrator's Deps.Storage now comes from the profile registry. Buildsignal keeps the queue partition (preserving per-queue serial ordering) but mints its speculate message ID from (batch, build) instead of reusing the bare batch ID, so the re-evaluation publish never collides with the batch controller's original message while a redelivered poll still dedupes against itself.

Test Plan

make e2e-test (submitqueue suite green — reproduces the Speculating stall without the message-ID fix) ✅ gateway + orchestrator integration suites via bazel ✅ go test ./...make fmtmake lintmake check-tidymake check-gazelle. Local docker-under-bazel runs need --sandbox_writable_path=$HOME/.docker for the image builds.

@behinddwalls
behinddwalls marked this pull request as ready for review August 6, 2026 17:01
@behinddwalls
behinddwalls requested review from a team and sbalabanov as code owners August 6, 2026 17:01
@behinddwalls
behinddwalls force-pushed the preetam/queue-leading-schema branch from 22d8314 to ecfcd55 Compare August 6, 2026 20:47
… fix the buildsignal republish dedup

## Summary

### Why?

Per-queue backend routing is host policy: profiles.go already routes every other seam (build runner, change provider, analyzer) per queue, so storage joins the same registry now that its contract is factory-resolved. Separately, end-to-end validation caught a regression from the payload change: repointing buildsignal's speculate publish from the batch-ID partition to the queue partition made its message share the (topic, partition, message-id) triple with the batch controller's original speculate publish, and the MySQL queue backend dedupes publishes on that triple against rows not yet garbage-collected — so the "build finished, re-evaluate" nudge was silently dropped and the batch stalled in Speculating.

### What?

`Profile` gains a per-queue storage factory entry; every profile points at the shared MySQL backend by default and `Profiles.StorageFactory()` routes each resolution through the queue's profile, so a deployment that splits queues across backends overrides one field. The orchestrator's `Deps.Storage` now comes from the profile registry. Buildsignal keeps the queue partition (preserving per-queue serial ordering) but mints its speculate message ID from (batch, build) instead of reusing the bare batch ID, so the re-evaluation publish never collides with the batch controller's original message while a redelivered poll still dedupes against itself.

## Test Plan

✅ `make e2e-test` (submitqueue suite green — reproduces the Speculating stall without the message-ID fix) ✅ gateway + orchestrator integration suites via bazel ✅ `go test ./...` ✅ `make fmt` ✅ `make lint` ✅ `make check-tidy` ✅ `make check-gazelle`. Local docker-under-bazel runs need `--sandbox_writable_path=$HOME/.docker` for the image builds.
@behinddwalls
behinddwalls force-pushed the preetam/storage-profile-routing branch from 29c8606 to 0cc7ace Compare August 6, 2026 20:49
@behinddwalls
behinddwalls changed the base branch from preetam/queue-leading-schema to main August 6, 2026 20:49
@behinddwalls
behinddwalls merged commit a15c40a into main Aug 6, 2026
3 checks passed
@behinddwalls
behinddwalls deleted the preetam/storage-profile-routing branch August 6, 2026 20:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants